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Background / Context: 

The consequences of failing core academic courses during the first year of high school are dire. 
Research by the Consortium on Chicago School Research (CCSR) demonstrates that students 
who fall “off track” during the first year of high school have a substantially lower probability of 
graduating than students who stay “on track.” In the Chicago Public Schools (CPS), only about 
one-fifth of off-track freshmen — students who fail more than one semester of a core academic 
course and/or fail to earn enough credits to be promoted to 10th grade — graduate high school, 
compared with over 80% of on-track freshmen (Allensworth & Easton, 2005, 2007). Failure of 
Algebra I is particularly problematic. In CPS, only 13% of students who fail both semesters of 
Algebra I in 9th grade graduate in 4 years, and the largest share of 9th grade algebra failures 
occur in the second semester of the course. Elucidating the ways that students can get back on 
track is of the utmost policy importance. 

Credit recovery is one strategy to deal with high failure rates. The primary goal of credit 
recovery programs is to give students an opportunity to retake classes that they failed in an effort 
to get them back on track and keep them in school (Watson & Gemin, 2008). Most recently, as 
schools across the nation struggle to keep students on track and re-engage students who are off 
track, online learning has emerged as a promising and increasingly popular strategy for credit 
recovery: more than half of respondents from a national survey of administrators from 2,500 
school districts reported using online learning in their schools for credit recovery, with just over 
a fifth (22%) reporting “wide use” of online learning for this purpose (Greaves & Hayes, 2008). 

Despite the growing use of online courses for credit recovery, the evidence base is thin. This 
paper describes the design, first year implementation, and early outcomes of a randomized 
control trial that was designed to strengthen the evidence base surrounding online courses used 
for credit recovery. 

Purpose / Objective / Research Question / Focus of Study: 

This study is an efficacy trial funded by a grant from the Institute of Education Sciences (lES) 
National Center for Education Research (NCER). Fifteen CPS high schools are receiving 
funding to implement two Algebra I credit recovery courses during the summer sessions of 201 1 
and 2012 — one online and one face-to-face (I2f). These courses allow students to recover a % 
credit of Algebra I. The study is designed to address a set of research questions that address the 
following specific aims: 

1) To test the efficacy of online Algebra I for credit recovery, compared with standard I2f 
Algebra I for credit recovery. 

2) To determine the supporting classroom conditions under which online Algebra I for 
credit recovery yields higher efficacy. 

3) To gauge the extent to which credit recovery can help at-risk students get back on track, 
relative to students who passed Algebra I in 9th grade. 

4) To gauge the effects of expanding summer credit recovery options through online 
courses. 
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In this paper, we will foeus on the impaet of taking online Algebra I for eredit reeovery on short- 
term outeomes for the first eohort of ninth-graders. These inelude eredit attainment in the eourse 
and seores on an end-of-eourse algebra test and on the PLAN assessment (a standardized pre- 
ACT taken in fall of grade 10). Over the eourse of this 4-year study, we will eontinue to follow 
the first eohort of students, as well as a seeond eohort, through high sehool to examine long-term 
effeets on future test seores, eoursetaking and likelihood of dropout. 

Setting: 

The setting for this study is CPS high sehools with the largest number of students who failed 
Algebra I. CPS is the third-largest U.S. distriet, serving more than 435,000 students in 666 
sehools, of whieh 116 are publie high sehools and 27 are publie eharter high sehools. Sehool 
reform and improvement have been high priorities in Chieago for a number of years, as high 
sehools in CPS eontinue to struggle with low student performanee and low graduation rates 
(Kahne, Sporte, de la Torre, & Easton, 2006). The overall graduation rate in the distriet is just 
54%. The average eomposite ACT seore for CPS juniors is 17, lower than the 20.5 for juniors in 
the state of Illinois and well below the seore required by most eolleges (Allensworth, Correa, & 
Poniseiak, 2008). 

Population / Participants / Subjects: 

The target students for this study were first-time freshmen who failed Algebra IB. Within eaeh 
sehool, the study team randomly assigned eligible students to eondition on site, on the first day 
of summer sehool. Students were eligible for random assignment if they failed seeond semester 
Algebra I, were willing to enroll in summer eredit reeovery, and showed up for elass on one of 
the first two days of the term. 

In year 1 (summer 2011) the study ineluded 18 pairs of online and 12f seetions of seeond 
semester algebra (Algebra IB) in 15 CPS high sehools. A total of 290 students were assigned to 
take the online eourse, and a total of 306 students were assigned to take the 12f eourse. Among 
the online students, 62% were male and 39% had passed first semester algebra during the 
freshman year. Among the f2f students, 65% were male and 38% had passed first semester 
algebra during their freshman year. 

Partieipating sehools tended to be larger than other high sehools in CPS — the average number of 
students enrolled in study high sehools was 1785 students, eompared to 729 students in non- 
study high sehools — with higher algebra failure rates. In other ways, they were similar to other 
sehools in the distriet. One exeeption is that the study sehools, on average, served a higher 
proportion of Hispanie students and students in whieh English was not the primary home 
language. 

Intervention / Program / Practice: 

The study is designed to test whether at-risk students who failed seeond semester Algebra I ean 
aehieve higher rates of sueeess if they take an online eourse for eredit reeovery than if they re- 
took the failed eourse in the standard 12f format. The theory of action behind this study is 
represented in Eigure 1 in Appendix B. Students fail algebra because they are poorly engaged in 
the class and put in little effort — the strongest predictors of 9th grade course failure are students’ 
attendance and work effort (Allensworth & Easton, 2007). Eow engagement leads students to 
learn little and to subsequently fail. Because they lack an understanding of algebra, they struggle 
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in subsequent classes, particularly in mathematics and science. Failure in these classes, combined 
with failure in algebra, leads students to have insufficient credits to graduate. As the likelihood 
of obtaining sufficient credits diminishes, students eventually drop out. 

Online credit recovery potentially interrupts this process in two ways. First, the delivery can be a 
more individualized, interactive experience. Furthermore, students receive personal support and 
monitoring from on-site mentors. These characteristics — individualization, interactive pedagogy, 
and personal support — have all been associated with greater engagement and learning 
(Archambault et ah, 2010; Lee & Smith, 1999; Newmann et ah, 1996; Slavin & Madden, 1989). 
Students should be more engaged and more likely to persist in the course, thus more likely to 
learn algebra content and receive course credit. These short-term outcomes should lead to 
improvements in other short-term achievement outcomes, including scores on the mathematics 
exam (that includes an algebra portion) taken in the fall of 10th grade. Better algebra skills 
should also make students more likely to pass their subsequent mathematics and science classes, 
and make greater progress toward graduation. 

The online course used in the study was developed by Aventa Learning, a provider that CPS has 
used extensively in recent years. Students took the course in computer labs at their local high 
schools, in the presence of a trained on-site mentor. They also had an online algebra teacher, 
provided by Aventa. The control condition is the typical f2f Algebra IB course offered in schools 
participating in the study. The course followed the standard CPS Algebra I curriculum and was 
taught by a teacher in each participating school. 

Research Design: 

The design of the study involves on-site student-level random assignment of students to either 
online or f2f Algebra I credit recovery courses in participating CPS high schools. The focus of 
student recruitment was on freshmen who failed second semester Algebra because they are 
positioned to get back on track in mathematics if they recover the 14 credit early on in high 
school. Students were blocked by gender and whether they passed or failed first semester 
Algebra and then randomly assigned to condition on site, on the first days of summer school. On- 
site random assignment was planned to prevent the inclusion of large numbers of no-shows in 
the intent-to-treat (ITT) analyses. 

In summer 2011, the study team randomly assigned a total of 596 students to condition (306 to 
online and 290 to I2f) across 36 sections (18 online and 18 f2f), yielding an average of 16.6 
students per section. Table 1 in Appendix B shows the distribution of students by and across 
condition on the blocking characteristics. 

The study was designed to have sufficient power to detect effects on student achievement and 
other outcomes ranging between 0.14 and 0.19 standard deviations, for analyses conducted 
separately by cohort. During the first summer of implementation, actual sample sizes were 
lower than the target sample sizes, but still provide adequate power to detect effects on student 
achievement outcomes. 



These minimum detectable effect sizes (MDESs) are reasonable based on previous research that finds the average 
effect of online learning relative to f2f is about 0.24 standard deviations (U.S. Department of Education, 2009). 
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This within-school, student-level randomized design establishes the core comparison that allows 
us to examine the impact of taking online Algebra I for credit recovery on students’ mathematics 
proficiency (score on end-of-course Algebra assessment and the PLAN) and credit attainment in 
Algebra I. 

Data Collection and Analysis: 

This paper will focus on outcomes based on the following measures: 

• A study-administered end-of-course assessment that includes a range of Algebra I 
topics. The assessment was administered to all consenting students participating in 
the credit recovery courses in summer 2011. 

• The PLAN assessment of the ACT Educational Planning and Assessment System 
(EPAS). Algebra subtest scores on the PLAN serve as a short-term achievement 
outcome because the assessment will be taken by students in the study within 3 
months of summer school, in fall 2011. PLAN scores will be collected for all cohort 
1 students from administrative records. 

• Performance in the summer 2011 credit recovery course, including course grades 
and credit attainment, collected via administrative records. 

The central research questions regarding the efficacy of online Algebra I for credit recovery 
versus Gf Algebra I for credit recovery are tested with 2-level models (with students nested 
within schools) that compare the outcomes of students in the online classes with those in the Gf 
classes. All analyses include the full ITT sample of online and Gf students. Analyses of 
achievement outcomes will employ hierarchical linear models controlling for student- and 
school-level covariates. Results will be reported both in their original metric and as effect sizes. 
Analyses of binary outcomes (i.e., whether students recovered the credit, or not) will employ 
hierarchical generalized linear models that assume a Bernoulli sampling distribution and logit 
link function (Raudenbush and Bryk 2002; McCullagh and Nelder 1998). Results will be 
reported in terms of differences in predicted probabilities (of participating, for example, in an 
advanced course sequence) and odds ratios. 

Findings / Results: 

The first year of implementation this study has been completed, and in this paper we will report 
on the effects of taking online Algebra I credit recovery compared with retaking Algebra I as a 
standard Gf summer course on students’ mathematics proficiency (scores on end-of-course 
assessment and on the PLAN) and credit attainment. The results are not yet available; we will be 
collecting student scores on the PLAN and on their credit attainment during the fall of 201 1 . All 
data will be collected and analyses will be completed in time for presentation at the SREE Spring 
2012 conference. 

Conclusions: 

The conclusions will be based on data from the first year of implementation. In short, we will be 
able to draw initial conclusions from this experimental study on the relative impact of online 
Algebra I credit recovery compared with a standard Gf algebra credit recovery course on 
students’ likelihood of recovering the credit and on their mathematics proficiency (particularly 
algebraic content). 
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Appendix B, Tables and Figures 



Figure 1, Theory of Action Behind Summer Online Algebra Credit Recovery 




Dashed lines indicate a negative relationship or disruption in the path. 



Table 1: Number and Percentage of Students Per Condition by Block, Summer 2011 







Passed Algebra lA 


Failed Algebra lA 


Algebra lA Status 
Unknown 


Total 


Condition 


Gender 


Number 


Percent of 
Students by 
Condition 


Number 


Percent of 
Students by 
Condition 


Number 


Percent of 
Students by 
Condition 




F2F 


Female 


44 


15% 


28 


10% 


31 


11% 


103 




Male 


70 


24% 


58 


20% 


59 


20% 


187 




Total 


114 


39% 


86 


30% 


90 


31%o 


290 


Online 


Female 


45 


15% 


37 


12% 


35 


11% 


117 




Male 


73 


24% 


60 


20% 


56 


18% 


189 




Total 


118 


39%, 


97 


32% 


91 


30% 


306 
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